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(54) Digital cameras 



(57) A method for a digital camera capturing tech- 
nique includes determining an orientation of an object 
(e.g., a document) relative to an optical axis of the digital 
camera, and determining a distance of the object from 
the digital camera The method also includes determin- 
ing a planarity of the object (e.g.. a curl of the document). 
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The method further includes projecting a pattern on the 
object, and detecting the projected pattern, which is then 
processed to determine the orientation of the object rel- 
ative to the optical axis of the digital camera, the dis- 
tance of the object from the digital camera, and the 
planarity of the object. 



00 

a> 
o 

Q. 
LU 



Printed by Jouve. 75001 PARIS (FR) 



1 



EP0 974 811 A1 



2 



Description 

[0001] The present invention relates generally to dig- 
ital cameras, and nnore particularly, to a digital camera 
capturing technique for documents. 
[0002] Projecting one or more points onto an object 
and detecting the projected points using an autofocus 
detector is the basis of so-called active (but nonacous- 
tic) autofocus mechanisms in typical point-and-shoot 
cameras. Cameras can reliably determine the focal dis- 
tance to a small area on an object, but cannot determine 
if an object is imaged obliquely, as can be the case with 
handheld exposures of documents. Cameras can indi- 
cate to the user that an object is too close to focus upon, 
but do not Indicate that the depth of focus is Inadequate 
to Image an entire object. Determining the adequacy of 
the depth of focus of an entire object represents a com- 
plex problem that is typically left to the user However, 
manual procedures, such as stopped-down viewfinder 
techniques used by sl^illed photographers, are very in- 
convenient and prone to error particularly in low-con- 
trast situations in a stopped-down viewfinder. 
[0003] According to a first aspect of the present inven- 
tion, there Is provided a method for a digital camera cap- 
turing technique, the method comprising: image sensor 
circultiy of the digital camera detecting an orientation of 
an object relative to an optical axis of the digital camera; 
and the image sensor circuitry capturing the object 
[0004] Emitter circuitry may project a pattern on the 
object, and image sensor circuitry may detect the pro- 
jected pattem. Multiple beams of light may be emitted, 
and the beams are preferably parallel. 
[0005] The multiple beams of light may comprise In- 
frared beams or diverging beams of light. 
[0006] The method may comprise storing In a memory 
data that represents the orientation of the object and the 
planarity of the object. 

[0007] According to a second aspect of the present 
invention, there is provided an apparatus for a digital 
camera capturing technique, the apparatus comprising: 
an Image sensor, the image sensor detecting a spot pat- 
tern projected on an object; and an image processor, 
the Image processor in communication with the image 
sensor, the image processor determining an orientation 
of the object relative to an optical axis of the digital cam- 
era, wherein the image sensor captures the object. 
[0008] The image processor may determine a planari- 
ty of the object and the apparatus may further comprise 
a plurality of emitters, the plurality of emitters projecting 
the spot pattern on the object. The image sensor may 
comprise a plurality of CMOS sensors and/or logic 
means that processes the projected spot pattern deter- 
mine the planarity and/or the orientation of the object, 
wherein the object connp rises a document. 
[0009] The image sensor may detect multiple beams 
of light projected on the object, wherein the object com- 
prises a document, whereby the image processor anal- 
yses a detection of the multiple beams of light projected 



on the document to determine an orientation of the doc- 
ument, the planarity of the document, a curl of the doc- 
ument, and a distance of the document. 
[0010] According to a third aspect of the present in- 

s vention. there is provided a method for a digital camera 
capturing technique for documents, the method com- 
prising: projecting a spot pattem on a document to be 
captured by the digital camera; detecting the projected 
spot pattem; analysing the projected spot pattern to de- 

10 tenmine an orientation of the document relative to an op- 
tical axis of the digital camera and a planarrty of the doc- 
ument. 

[0011] Data representing the orientation and planarity 
of a document may be processed prior to capturing one 

1^ "document, whereby the image sensor captures the doc- 
ument in sharp focus and without geometric distortions. 
Multiple exposures of the document may be processed 
to capture portions of the document that are at multiple 
distances in sharp focus. The method may further com- 

20 prise outputting a signal to indicate that the digital cam- 
era should be reoriented with respect to the document. 
[0012] The projected spot pattern may be analysed to 
determine a distance of the document from the digital 
camera. 

25 [001 3] In one embodiment, the present invention is in- 
corporated Into a digital camera to permit compensation 
to be performed for the following effects in the handheld 
capture of documents at close range: 

50 • the optical axis of the camera is not aligned with a 
normal to the surface of the document; 

• the depth-of-field for optics used for capture within 
0.5 - 1 meter from the focal plane of the Image sen- 

^ sor is limited, so proper focus is critical; and 

• typical documents may not be planar, for example, 
pages bound In a book. 

This embodiment extends the concept of auto-focus for 
cameras to auto-focus and auto-orlentation, and pro- 
vides an indication of the orientation of the optical axis 
with respect to the normal to the plane of the document 
and an indication of the degree of planarity of the doc- 
ument (e.g., including page curi of the document). 
[0014] A typical active autofocus camera employs an 
infrared emitter and a photodetector arranged on either 
side of the optical axis. As the shutter release is pressed 
part-way, a highly-collimated IR beam Is projected (usu- 
ally as a vertical bar) on the subject, and the position of 
the beam is imaged on the detector. The measurement 
of range relies on parallax between the emitter and de- 
tector. 

[0015] In one embodiment, the present invention in- 
volves projecting beams of visible or, preferably, infrared 
(IR) light on the document to be captured in a predeter- 
mined pattem (e.g., a spot pattem) as the shutter re- 
lease is pressed part-way. The projected pattem is de- 
signed to provide information of range, orientation, and 
planarity of the document. The pattem is received by the 
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image sensor in the digital camera and stored (or proc- 
essed) immediately before the capture of the subject im- 
age using the image sensor. Using IR illumination pro- 
vides an invisible measurement and uses the sensitivity 
of, for example, commercially available CCD and CMOS 
sensors to IR wavelengths. High intensity IR-LEDs are 
commercially available for use as emitters. 
[0016] A pattern to provide range and alignment can 
be provided by projecting three beams of light in a pat- 
tern. More robust patterns for measuring planarity of the 
subject can employ more than three beams of light, for 
example, to provide a grid of multiple cells. 
[001 7] In one embodiment, a method for a digital cam- 
era capturing technique includes determining an orien- 
tation of an object (e.g.. a document) relative to an op- 
tical axis of the digital camera, and determining a dis- 
tance of the object from the digital camera. The method 
also includes determining a planarity of the object. The 
method further includes projecting a pattem on the ob- 
ject, and detecting the projected pattern, which is then 
processed to determine the orientation of the object rel- 
ative to the optical axis of the digital camera, the dis- 
tance of the object from the digital camera, and the 
planarity of the object. 

[0018] Other aspects and advantages of the present 
invention will become apparent from the following de- 
tailed description and accompanying drawings. 
[0019] FIG. la shows a regular S-by-S array of spots 
of light (a spot pattern) projected on an object, and FIGs. 
1 b-1 e show examples of the spot pattern for various ob- 
ject orientations and positions in accordance with one 
embodiment of the present invention. 
[0020] FIGs. 2a-2e are similar to FIGs. 1 a-1 e but with 
a spot pattern that includes only four spots in accord- 
ance with another embodiment of the present invention. 
[0021] FIGs. 3a and 3b present an imaging approach 
for detecting distance and orientation of the object to be 
captured in accordance with one embodiment of the 
present invention. 

[0022] FIG. 4 presents an imaging approach for de- 
tecting distance and orientation of the object to be cap- 
tured in accordance with another embodiment of the 
present invention. 

[0023] FIG. 5 shows a digital camera in accordance 
with one embodiment of the present invention. 
[0024] FIG. 6 is a flow diagram of the operation of the 
digital camera of FIG. 5 in accordance with one embod- 
iment of the present invention. 

[0025] FIG, 7 is a block diagram of the digital camera 
of FIG. 5 shown in greater detail in accordance with one 
embodiment of the present invention. 
[0026] There are many usability issues and technical 
limitations in handheld document imaging. Some of the 
current generation of digital cameras have a document 
Capture mode, but the low pixel count in the image sen- 
sors of the current generation of digital cameras makes 
them useful only for small documents, such as business 
cards. Howevei^, In the future, affordable digital cameras 



will likely have more than four million pixels, which will 
allow for full-page documents to be captured at a high 
level of document quality. 

[0027] Accordingly, the present inventbn provides a 

5 digital camera capturing technique for documents. In 
one embodiment, the present invention uses projected 
beams of light (e.g., a spot pattern) and an image sensor 
of a digital camera to determine a distance, an orienta- 
tion, and a planarity of an object to be captured. It is 

10 assumed that the spots are sharply imaged on the im- 
age sensor, which generally requires that the camera's 
autofocus system provide a useful, approximate focal 
distance measurement so that camera optics can be ad- 
justed for optimal focus at close range. An object detect- 

is ed at close range automatically activates the document 
capture mode with its beam emissions and processing 
described herein. Measurement of the intensity distribu- 
tion for each imaged spot provides an indication of the 
sharpness of focus and the usefulness of the spot for 

^0 measurement purposes. 

[0028] FIG. la shows a regular 5-by-5 array of spots 
of light projected on an object in accordance with one 
embodiment of the present invention. It is not necessary 
to have a regular array so long as the arrangement of 
the spots is taken into account when processing the ac- 
quired image of the spots on the object to be captured. 
FIGs. 1b-1e show examples of the spot pattern of FIG. 
1 a for vark>us object orientations and positions in accord- 
ance with one embodiment of the present invention. 

30 [0029] RG. 1 b shows an example innage of the spot 
pattern when the object is pitched and rolled with re- 
spect to the optical axis. 

[0030] FIG. 1c shows an example image of the spot 
pattern when the object is pitched with respect to the 

35 optical axis. 

[0031] FIG. Id shows an example image of the spot 
pattern where the optical and normal axes are coinci- 
dent near the right-half of the object, but it is a book 
which is nonplanar, and the spots on the left indicate 

40 curvature of the object out of the plane, which is often 
referred to as page curl. 

[0032] FIG. 1 e is another example of the image of the 
spot pattem projected onto an open, bound book where 
there is pitch of the surface as well as curvature. 
45 [0033] FIGs. 2a-2e are similar to FIGs. 1 a-1 e but with 
a spot pattern that includes only four spots in accord- 
ance with another embodiment of the present invention. 
[0034] FIGs. 3a and 3b present an imaging approach 
for detecting distance and orientation of the object to be 
50 captured in accordance with one embodiment of the 
present invention. We assume the following for the sake 
of illustration of the general principle: 

• The object is a predominantly planar surface whose 
55 normal vector has an arbitrary angular rotation with 

respect to the camera's optical axis. 

• The rotation of the plane is about a vector perpen- 
dicular to the plane of emitters 1 A and 2A in FIGS. 
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3a and 3b. 

• Beams 1 and 2 are parallel to the optical axis and 
thereby form a plane containing all the points indi- 
cated in FlGs. 3a and 3b. This parallelism is not a 
requirement for this technique to work (e.g., FIG. 4 
illustrates the use of diverging beams). 

[0035] In a general case, the rotation of the plane is 
about an axis that is not perpendicular to the plane 
formed by beams 1 and 2. Some curvature of the object 
surface is allowed, for example, pages in a bound book 
and documents printed upon or wrapped around a cyl- 
inder. 

[0036] In FIG. 3a, emitters 1 A and 2A project beams 
1 and 2 parallel to the optical axis. Object orientation A 
shows a plane rotated with respect to the optical axis 
(e.g., an oblique parallel projection). Beams 1 and 2 pro- 
duce spots 1 B and 2B, respectively. The images of spots 
1 B and 2B projected by the camera's lens onto Its image 
sensor are 1C and 2C, respectively. 
[0037] The orientation and distance of the plane can 
be determined from the distances 1-1 C (the distance be- 
tween two points A and B is denoted as A-B) and I-2C. 
These distances can be measured In pixels on the im- 
age sensor in the camera's image plane. The inequality 
of I-1C and I-2C indicates that the plane is pitched with 
respect to the optical axis. 

[0038] When the plane is rotated to align its normal 
with the optical axis, beams 1 and 2 produce spots IB 
and 2D. respectively, for example, in object orientation 
B, which are imaged as points 1C and 2E, respectively. 
In this case, I-1C and l-2E are identical, indicating align- 
ment of the normal vector and the optical axis. 
[0039] FIG. 3b demonstrates distance measurement 
when the plane is perpendicular to the optical axis and 
translated between two positions (e.g., an orthographic 
parallel projection). In position A, beams 1 and 2 pro- 
duce spots 1B and 2B on the object that are imaged as 
1 0 and 2C, respectively. The distance to the object can 
be computed from 1-1 C and I-2C. Moving the object far- 
ther away, beams 1 and 2 produce spots 1 D and 2D that 
are imaged as 1 E and 2E, respectively. Because the dis- 
tance JMC is greater than 1-1E. the object in position B 
Is determined to be further away than the object in po- 
sition A. 

[0040] In this example, there Is a single rotation of the 
object's normal with respect to the imaging axis, and 
beams 1 and 2 are parallel to the optical axis. The cal- 
culation of distance and orlentatbn of the object repre- 
sents an application of planar trigonometry using 
known, fixed distances in the projection and imaging 
system and the position of the imaged spots. 
[0041] The distance between points 1 A and 1 B, tA; 
1B, is computed as follows. It is assumed, in this exam- 
ple, that emitters 1 A and IB are located in the image 
plane (which introduces a constant in the calculation 
that can be omitted for clarity). By similar triangles, we 
know that 



1A-1B /I-F = 1A-1C / MC . (1 ) 

Solving for the only unknown, 

5 

.1A-1B = 1A-1C * (I-F/I-1C) (2a) 

and using the relation between fixed distances 1A-1C = 
MA + Ma 

1A-1B = I-F * ( I-1A / I-1C ). (2b) 
Similarly, it can be shown that 

2A-2B = l;f * ( I-2A/I-2C ) (3) 

20 

[0042] An optimal focal distance for document cap- 
ture can be computed from 1 A-1 B and 2A-2B taking into 
account the depth of fiekl for the lens at its working ap- 
erture. 

[0043] The situation of a rotated object places de- 
mands on the camera's ability to deliver sharp focus 
across the document: for example, portions of the doc- 
ument may lie outside the camera's depth -of -focus. In 
this case, information obtained by the present invention 
can be used to nnake multiple exposures, each at differ- 
ent working distances, in order to capture the document 
piece-wise in sharp focus. Altematively, the Information 
that the camera's depth -of -focus is exceeded could be 
used to generate a signal to inform the user to reorient 
the camera with respect to the document. This informa- 
tion, which typically cannot be determined from a single 
autofocus measurement at a point on a document, pro- 
vides a valuable feature for opportunistic and casual 
capture of documents where precise alignment between 
the camera and the object cannot be or is not obtained. 
[0044] the angle 9 that the plane makes with the op- 
tical axis in object orientation A in FIG. 3a is 

arcsin (( 2A-2B - 1A-1B ) /1A-2A ). (4) 

IS 

[0045] The constructions in FIGs. 3a and 3b and 
equations (1) - (4) show that, for parallel beams. It is 
desirable to maximize the offsets of the emitters from 
the optical axis (i.e., 1-1 A I-2A) to achieve the maxinrium 
pixel count for 1-1 C and I-2C. Achieving the maximum 
pixel count for 1-1 C and U2C provides the greatest res- 
olution in equations (2b), (3), and (4) given the discrete 
number of pixels. 

[0046] In the general case, where the rotation of the 
object is along a vector that is not perpendicular to the 
plane of emitters 1 and 2, more complex geonrietric 
transforms than equatk>ns (1) - (4) are required and 
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would be apparent to one of ordinary skill in the art. 
[0047] Planarity (e.g.. distortion) of an object to be 
captured is determined, for example, using a spot pat- 
tern that includes more than 4 spots and by applying the 
above discussed logic to such a spot pattern. "Digital s 
Image Warping", George Wolberg, IEEE Computer So- 
ciety 1 990, pp. 52-56, discusses the principle of a 4-dot 
matrix for perspective transformations, and is incorpo- 
rated herein by reference in its entirety. 
[0048] FIG. 4 shows another embodiment In which io 
emitters 3A and 4A are placed together on the optical 
axis and project a pattern using non-parallel (diverging) 
beams aimed toward the object. The non-parallel 
beams produce spots 3B and 4B on a rotated object. 
The Images of these points at the camera's focal plane is 
sensor are 3C and AC, respectively 
[0049] In particular, FIG, 4 illustrates the different re- 
sults for parallel beams l and 2 and divergent beams 3 
and 4. It is observed that the difference I-3C - I-4C is 
small for two significantly different distances 3A-3B and 20 
4A-4B. This observation suggests that the co-located, 
diverging beam method has less sensitivity and greater 
susceptibility to discretization errors than the parallel 
beam method, where the difference 1-1 C - i-2C generally 
provides greater sensitivity. 2S 
[0050] FIG. 5 shows a digital camera 50 in accord- 
ance with one embodiment of the present invention. Dig- 
ital camera 50 includes IR emitters 1 A, 1 A', 2A, and 2A' 
arranged around the optical axis of the camera's imag- 
ing system at a displacement that represents a reason- 30 
able compromise between usability and sensitivity. Dig- 
ital camera 50 also includes an active autofocus emitter 
51 , a viewfinder 52. an autofocus detector 53, an orien- 
tation/range emitter 54. and an imaging lens 55. 
[0051] FIG. 6 Is a flow diagram of the operation of the 3S 
digital camera of FIG. 5 in accordance with one embod- 
iment of the present invention. As the shutter release of 
digital camera 50 is pressed, digital camera 50 projects 
a spot pattern using IR emitters 1 A, 1 A', 2A. and 2A' to 
emit parallel IR beams on the object at stage 61 . Digital "to 
camera 50 detects the spot pattem on its image sensor 
(see FIG. 7) at stage 62. Digital camera 50 determines 
the orientation of the object at stage 63, as discussed 
above with respect to FIGs. 3a and 3b. Digital camera 
50 determines the planarity of the object at stage 64, as 
discussed above with respect to FIGs. 3a and 3b. Digital 
camera 50 can also determine the distance of the object 
from digital camera 50. as discussed above. Further, 
digital camera 50 can determine a (page) curl of the ob- 
ject, for example, In which the object Is a document such so 
as a page in a bound book. 

[0052] Digital camera 50 then processes this informa- 
tion accordingly, as discussed above with respect to 
FIGs. 3A and 3B. Alternatively, digital camera 50 can 
emit non-parallel beams of light and perfomrt stages 63 ss 
and 64 as discussed above with respect to FIG. 4. For 
example, output from orientation, planarity, and page 
curl determinations can be used by Image processing 
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algorithms to render a planar, undistorted image of a 
document captured obliquely and with page curl (e.g.. 
a page of a bound book). 

[0053] FIG. 7 is a block diagram of the digital camera 
of FIG. 5 shown in greater detail in accordance with one 
embodiment of the present invention. In FIG. 7, digital 
camera 50 includes an ASIC 72 (Application Specific In- 
tegrated Circuit) that is in communication with orienta- 
tion/range emitters 1 A 1 A', 2A, and 2A', active autofo- 
cus emitter 51 , viewfinder 52, autofocus detector 53. a 
memory 76 (e.g.. a conventional random access mem- 
ory that is used to store a captured digital image of a 
document) via a bus 74, and an image sensor 78 (e.g., 
a photodetector array that Includes CMOS sensors for 
detecting IR light emitted from emitters 1 A, 1 A', 2A, and 
2A*). For example, image sensor 78, which is used for 
image capture, is first used to acquire information relat- 
ing to the distance, orientation, planarity, and curl of the 
document to be captured using the projected sp)ot pat- 
tern (e.g., obtaining real-time Information from a pre-ex- 
posure). 

[0054] ASIC 74 represents a main ASIC of the digital 
camera. ASIC 74 performs image processing. For ex- 
ample, the image processing involved In the present in- 
vention can be Implemented on image processing ASIC 
74. For example, the logic for performing the digital cam- 
era capturing technique can be implemented in a hard- 
ware description language, such as the Verilog^ hard- 
ware description language, described, for example. In 
D.E. Thomas. J. P. Moorby, "The Verllog™ Hardware 
Description Language' (1991). Thus, the logic for per- 
forming the digital camera capturing technique can be 
Implemented in the Verllog"™ hardware description lan- 
guage, and the corresponding circuitry can be Imple- 
mented using methods known in the art to provide an 
image processing ASIC for performing the digital cam- 
era capturing technique in accordance with one embod- 
iment of the present Invention (e.g., an ASIC Implemen- 
tation including logic that performs 3D geometrical 
transformations using a lookup table for Inverse trigono- 
metric calculations). 

[0055] In one embodiment, the present invention in- 
cludes the following: 

• providing physical measurements describing the 
distance, shape (e.g., planarity, such as curl of a 
document), orientation of an object (e.g., a docu- 
ment), and curi of the object to be captured with re- 
spect to the optical axis of a camera; 

• providing measurements of the distance, shape, 
orientation, and curl of a document to an irDage 
processor such that the document can be rendered 
as a planar object without geometric distortions; 

• provides an indication of the angular-orientation of 
a camera's optical axis with respect to a normal to 
the surface of an object; 

• provides an Indication of the planarity of an object; 

• operates In conjunction with the camera's active or 
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passive range-finding system to detect a close-fo- 
cus situation and enhance the capture of docu- 
ments at close focal distances; 

• provides a signal to the camera's exposure archi- 
tecture that multiple exposures at different focal dis- 
tances are required to render the document given 
limitations on the camera's depth of focus in close- 
focus situations; and 

• provides an indication to the user that the camera 
must be reoriented with respect to the document to 
properly capture the document. 

[0056] Although particular embodiments of the 
present invention have been shown and described, it will 
be obvious to those skilled in the art that changes and 
modifications can be made without departing from the 
present invention in its broader aspects. For example, 
the present invention can be Implemented In hardware 
or software or a combination of hardware and software. 
The present invention can also be implemented using 
any number and arrangement of orientation/range emit- 
ters providing a variety of spot patterns for Image 
processing. The image sensor circuit can be implement- 
ed using CMOS sensors. CCD sensors, or other types 
of sensors that can detect a projected spot pattern. 
Therefore, the appended claims are to encompass with- 
in their scope all such changes and modifications that 
fall within the true scope erf the present invention. 



Claims 

1. A method for a digital camera capturing technique, 
the method comprising: 

image sensor circuitry of the digital camera de- 
tecting an orientation of an object relative to an 
optical axis of the digital camera; and 
the image sensor circuitry capturing the object. 

2. The method of claim 1 further comprising: 

the image sensor circuitry detecting a planarity 
of the object. 

3. The method of claim 1 further comprising: 

the image sensor circuitry detecting a curl of the 
object, wherein the object comprises a docu- 
ment. 

4. The method of claim 1 further comprising: 

emitter circuitry projecting a pattern on the ob- 
ject. 

5. The method of claim 4 further comprising: 



the image sensor circuitry detecting a projected 
pattern. 

6. The method of claim 5 further comprising: 

5 

image processing circuitry processing a detec- 
tion of the projected pattern to determine the 
orientation of the object and the planarity of the 
object, wherein the object comprises a docu- 
ment. 

7. The method of claim 5 wherein the projected pattem 
comprises at least four projected spots. 

8. The method of claim 5 wherein the projecting pat- 
tem comprises emitting multiple beams of light. 

9. An apparatus for a digital camera capturing tech- 
nique, the apparatus comprising: 

20 

an Image sensor, the image sensor detecting a 
spot pattern projected on an object; and 
an image processor, the inr^ge processor in 
communication with the image sensor, the im- 
age processor determining an orientation of the 
object relative to an optical axis of the digital 
camera, wherein the image sensor captures 
the object. 

30 10. A method for digital camera capturing technique for 
documents, the method comprising: 

projecting a spot pattem on a document to be 
captured by the digital camera; 
3^ detecting the projected spot pattern; 

analysing the projected spot pattern to deter- 
mine an orientation of the document relative to 
an optical axis of the digital camera and a 
planarity of the document. 
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